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(54) Extending the vocabulary of a client-server speech recognition system 



(57) A network spoken language vocabulary system 
for a speech application comprises: a network server 
(12) and a plurality of network clients (14, 16, 18) com- 
municating with one another over a network; a central 
vocabulary list (24) in the server for recognizable words; 
and a speech recognition engine (32) and a local vocab- 
ulary list cache (36) in each of the clients. The cache 
(36) has therein previously recognized words commu- 



nicated from the central vocabulary list (24) in the server 
and new words recognized by the speech application. 
Each of the new words is communicated to the server 
and added to the central vocabulary list and each of the 
new words added to the central vocabulary list is com- 
municated to at least one other of the clients for addition 
to the cache. The new words can be automatically com- 
municated to and from the server. 
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Description 

Background of the Invention 

1 . Field of the Invention 

[0001] This invention relates generally to the field of 
speech applications, and in particular, to a method and 
apparatus for automatically storing, tracking and distrib- 
uting new word pronunciations to speech application cli- 
ents on a network. 

2. Description of Related Art 

[0002] Use of spoken language with computers, typi- 
cally associated with speech recognition and speech 
synthesis, involves storing and retrieving not only word 
spellings but other data associated with words, for ex- 
ample phonemes, alternate pronunciations, associa- 
tions with other words and parts of speech, such as 
noun, verb, adjective and the like. 
[0003] Computer systems at first were developed to 
deal exclusively with written language. Written language 
is useful for many things, and is much simpler to inter- 
pret, represent within, and reproduce from a computer 
system than is spoken language. Today, computer sys- 
tems are taking on spoken language in the form of two 
technologies, namely, speech recognition and speech 
synthesis. Speech synthesis is also referred to as text- 
to-speech (TTS). 

[0004] Defining the work to be done by computers to 
support spoken language is aided by comparing spoken 
language to written language. In the intersection of 
these two forms of communication there are words. Out- 
side of the intersection words are represented different- 
ly, as spoken sounds or written letters. Written language 
is also augmented outside of the intersection by punc- 
tuation, or even font variations such as bold for empha- 
sis. Spoken language is augmented differently outside 
of the intersection, for example by volume, pitch, pros- 
ody (speed), and inflection. 

[0005] As computers tackle support for spoken lan- 
guage, spoken language as typically spoken is convert- 
ed to text form by way of speech recognition and con- 
verted back to spoken form by way of speech synthesis. 
This takes advantage of the significantly reduced re- 
quirements on system resources to store or transmit a 
written representation as compared to an audible rep- 
resentation. The differences between written and spo- 
ken words outside of the intersection create a number 
of problems for speech applications. 
[0006] End-users are greatly inconvenienced by the 
need to add word pronunciations which are not included 
in the starter set of vocabulary words which can be rec- 
ognized. By design, the user encounters this problem 
as a special case of a word which was recognized in- 
correctly. Recognition can be viewed as a best-guess 
by the recognition engine as to which word was spoken 
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by the user. When the user speaks a word which is not 
known to the recognition engine, the engine simply 
guesses wrong. The user must then initiate correction 
of the word and choose a new word from a short list of 
5 appropriate alternatives. If the spoken word is not listed 
as an alternate choice, the user typically is required to 
type the word and perhaps pronounce it again. This in- 
convenience can encourage users to bypass the proper 
correction procedures and simply type in the corrected 
10 text. Unfortunately, although this procedure is quicker in 
the short run, it is important that the speech recognition 
system learn about the correction with the proper pro- 
cedures because this information is the only way to add 
and correct words, thereby improving future recognition 
*s performance. 

[0007] Speech recognition engines as supplied in 
speech applications just aren't accurate enough and are 
sometimes slow in throughput due to misrecognitions 
and the time needed to correct misrecognitions. 
20 [0008] Correcting misrecognized words is a major 
factor in measures of speech recognition effectiveness 
including words-per-minute and usability. Large vocab- 
ularies are provided to limit the number of corrections 
resulting from out-of -vocabulary words. 
25 [0009] In a stand alone system, user added words 
need to be backed up and migrated from system to sys- 
tem as a user moves around or switches between sys- 
tems (for example, between home and office computers 
and between portable and desktop computers) or up- 
30 grades a computer or a speech recognition application, 
or both. This task is time consuming, tedious, unobvi- 
ous, and subsequently, not generally done. 
[0010] Typically, along with speech recognition or 
synthesis software, a starter set of words, including pro- 
35 nunciations, is installed on a computer. Pronunciations 
are stored as base forms, which represent instructions 
as to the way words are pronounced, or sound. Many 
factors must be considered in order to produce an ideal 
starter set for a given application. The number of words 
40 jn a starter set is usually determined by balancing con- 
siderations such as amount of storage space required, 
frequency of word use, degree of common use, and rec- 
ognition accuracy against all other words in the starter 
set. Developers of speech recognition systems typically 
^5 install a large vocabulary of word-pronunciation data as 
required for recognition. It is a rather tedious and time 
consuming task to produce such a vocabulary. 

Summary of the Invention 

so 

[0011] A network universal spoken language vocabu- 
lary developed and managed in accordance with the in- 
ventive arrangements overcomes all of the problems 
noted in the prior art. The inventive arrangements auto- 
55 mate and optimize storage, tracking, sharing, and deliv- 
ery of spoken language vocabulary data. Word informa- 
tion regarding all known words can be distributed from 
a network connected server. New words added by any 



2 



3 



EP 0 945 851 A2 



4 



client user are made available to all users, eliminating 
the need for subsequent users to each provide data re- 
garding that same new word. 

[0012] Fewer manual additions of pronunciations are 
required from each user, so that the users are more like- 
ly to follow those procedures for correcting misrecogni- 
tions in a manner which teaches the speech application 
new words, thus improving future recognition perform- 
ance for all users. 

[001 3] A speech recognition and/or speech synthesis 
system or application according to the inventive ar- 
rangements eliminates the need for more than one user 
to add a new word. Only the very first user of a new word 
needs to add that new word. The system can, while rec- 
ognizing or determining alternatives, query a universal 
vocabulary server for a new word and will not bother the 
user to provide the data. 

[001 4] Increased accuracy and throughput of speech 
recognition are among the advantages of a highly opti- 
mized, cached universal vocabulary simply by virtue of 
a higher hit-rate between words spoken and words 
known and accurately recognized. 
[0015] The requirements for installing and maintain- 
ing a large vocabulary on each system is advantageous- 
ly reduced. It is transparent to the user that some words 
are stored on a server, and as the system gets used, 
caching will increase the hit-rate, reducing the number 
of words stored on a client as well as generally speeding 
things up. 

[0016] The need to maintain a users added words, 
and the need to individual back up and migrate updated 
vocabulary lists is advantageously eliminated by cen- 
tralized data backup. 

[0017] The burden of carefully selecting the most ap- 
propriate starter set in advance is advantageously elim- 
inated because specialized words will be added effi- 
ciently and in a shared-burden manner by the network 
of users. On the contrary, the vocabulary for all users is 
automatically updated as needed, without requiring user 
interaction. 

[0018] A network spoken language vocabulary sys- 
tem for a speech application, in accordance with an in- 
ventive arrangement, comprises: a network server and 
a plu rality ol network clients communicating with one an- 
other over a network: a central vocabulary list in the 
server for recognizable words; a speech recognition en- 
gine and a local vocabulary fist cache in each of the cli- 
ents, the cache having therein previously recognized 
words communicated from the central vocabulary list in 
the server and new words recognized by the speech ap- 
plication; and : each of the new words being communi- 
cated to the server and added to the central vocabulary 
list and each of the new words added to the central vo- 
cabulary list being communicated to at least one other 
of the clients for addition to the cache. 
[0019] The new words can be automatically commu- 
nicated from the clients to the server and can also be 
automatically communicated from the server to one or 



more clients. 

[0020] The server can further comprise a user profile 
table having data therein for each of the users indicative 
of a respective subset of the central vocabulary list for 

5 use in speech recognition data and/or indicative of 
membership in a work group. When this data is availa- 
ble, each new word communicated to the server from 
any user in one of the workgroups can be automatically 
communicated back to all other users in the work group. 

io Moreover, each user in the work group can receive the 
same subset of the central vocabulary list for use in 
speech recognition. 

[0021] Each of the clients can further comprise an add 
word pronunciation tool for identifying the new words to 
75 be communicated to the server, and for adding the new 
words to the cache. The add word pronunciation tool can 
add the new words as recognized by the speech recog- 
nition engine in the client and as communicated from 
the server. 

20 [0022] The central vocabulary list can comprise a plu- 
rality ol data records for each word, the data records 
including: spelling; phonemes; usage domains; client of 
origin; and, workgroups requiring the word. 
[0023] A network spoken language vocabulary sys- 

2S tern for a speech application, in accordance with another 
inventive arrangement, comprises: a network server 
having a permanent central vocabulary list for recogniz- 
able words and a user profile data base; a plurality of 
network clients, each having a speech recognition en- 

30 gine and a local vocabulary list cache; a network over 
which the server and the clients can communicate; the 
cache having stored therein a subset of previously rec- 
ognized words communicated from the central vocabu- 
lary list in the server and new words as recognized by 

35 the speech application during the session; each of the 
new words being communicated to the server and add- 
ed to the central vocabulary list; and, each of the new 
words added to the central vocabulary list being com- 
municated to other ones of the clients in accordance 

40 with data in the user profile data base. 

[0024] The new words can be automatically commu- 
nicated to and from the server. 

[0025] The user profile data base has data therein for 
each of the users indicative of a respective subset of the 

45 central vocabulary list for use in speech recognition and/ 
or indicative of membership in a work group. When this 
information is available, each new word communicated 
to the server from any user in one ol the workgroups can 
be automatically communicated back to all other users 

50 in the work group. Moreover, each user in the work 
group can receive the same subset of the central vocab- 
ulary list for use in speech recognition. 
[0026] Each of the clients can further comprise an add 
word pronunciation tool for identifying the new words to 

55 be communicated to the server, and for adding the new 
words to the cache. The add word pronunciation tool can 
add the new words as recognized by the speech recog- 
nition engine in the client and as communicated from 
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the server. 

[0027] The central vocabulary list can comprise a plu- 
rality of data records for each word, the data records 
including: spelling; phonemes: usage domains: client of 
origin; and, workgroups requiring the word. 5 

Brief Description of the Drawings 

[0028] The sole Figure is a block diagram of a network 
universal spoken language vocabulary system in ac- io 
cordance with the inventive arrangements. 

Detailed Description of the Preferred Embodiments 

[0029] A comprehensive system 10 which covers all is 
facets of vocabulary support for spoken language tech- 
nologies such as speech recognition and speech syn- 
thesis is shown in the sole Figure. The system compris- 
es a word pronunciation or vocabulary server 12 con- 
nected to a plurality of vocabulary clients 14, 16 and 18 20 
over a network 20. The network 20 can be a local area 
network (LAN) or a wide area network (WAN). In accord- 
ance with the inventive arrangements, the word pronun- 
ciation or vocabulary server 12 is a universal repository 
for all known words and associated spoken language 25 
attributes. New words are acquired from and distributed 
to network clients as needed. 

[0030] The word pronunciation server 12 comprises 
a universal vocabulary server program 22, a universal 
vocabulary list, or data base 24 and a user profile table 30 
26. The word pronunciation server 12 is a network con- 
nected computer system which hosts the universal vo- 
cabulary database and the universal vocabulary server 
software program 22 which responds to requests by cli- 
ent computer systems which are running speech recog- 35 
nition or speech synthesis programs. 
[0031] The universal vocabulary list 24 includes all 
words known to the system including the spelling and 
data indicating how the word is pronounced. The term 
universal denotes throughout the network. In this sense, 40 
the universal vocabulary list can also be thought of as 
a central or master vocabulary list. Data records for each 
word entry include the following information: spelling; 
phonemes, that is, pronunciation data; list of usage do- 
mains; origin, that is, who added the word; and, work- 45 
group, that is, which users are likely to use the word. 
[0032] The user vocabulary profile 26 is a database 
which contains information required to support the uni- 
versal vocabulary client program. Data regarding which 
word groups a particular user uses, and the user groups so 
which the user belongs to is maintained on the server 
and transmitted to each client when they begin a session 
with speech recognition or TTS software. 
[0033] The term client is used in the sense of a work 
station served by the word pronunciation server. Users 55 
are persons working at the clients. The user profile table 
or data base 26 can be keyed to users, who can be iden- 
* tified for example by name or by log in identification, so 



that any user can be provided with the appropriate sub- 
set of vocabulary words irrespective of the particular 
work station or client being utilized. If the user of a par- 
ticular client or work station is not in the user profile ta- 
ble, the user profile table can also be keyed on the client 
or word station identification. Alternatively, a default 
subset for unidentified users can also be defined. As a 
further and presently preferred alternative, an unidenti- 
fied user can be prompted to supply sufficient informa- 
tion to be identified in the user profile table, and there- 
after, be supplied with an appropriate subset vocabulary 
list and appropriate new words. 

[0034] The proper vocabulary list can even be trans- 
ferred to a remote work station by modem, satellite 
transmission, or the Internet, although immediate and 
automatic updates would be available only for so long 
as the remote link was maintained. In this respect, the 
network 20 also represents remote links. 
[0035] Each of the clients 1 4, 1 6 and 1 8 comprises a 
universal vocabulary client 30, a speech recognition en- 
gine 32, an add word pronunciation tool 34 and a local 
vocabulary list cache 36, for storing a personalized sub- 
set of the universal vocabulary list 24. For purposes of 
illustration, clients 14 and 16 are members of a work 
group associated with a user profile designated A. Client 
18 is a member of a work group associated with a user 
profile designated B. 

[0036] The add word pronunciation tool 34 transmits 
word newly added to each client over the network 20 to 
the word pronunciation server 12, where the new words 
are added to the universal vocabulary list 24. The add 
word pronunciation tool 34 also receives new words 
from the word pronunciation server 12 over the network 
20, and routes the new words to the local list cache. 36. 
[0037] The universal vocabulary client program 30 is 
an application which runs on a client system maintaining 
the local cached subset 36 of the universal vocabulary 
list 24. Words are added to the cache from the user and 
from the universal vocabulary list, while words which 
have expired in terms of cache -worthiness are deleted. 
The cache 36 is a dynamically optimized replacement 
for the traditional stand-alone starter-set vocabulary. 
[0038] The programs which maintain the cache 36 
employ algorithms which can consider network traffic, 
available local storage, frequency of use : system proc- 
essor speed, current end-user task, and end-user set- 
tings to determine the parameters for the cache. The 
cache can be thought of as an optimal local vocabulary. 
The cache is optimal because specialized words known 
to be spoken by the user will be included, enhancing 
recognition, and at the same time, words known not to 
be spoken by the user will not be included, reducing 
processing time needed to search the cache for match- 
es. 

[0039] The client application sends data regarding 
newly added words to the universal vocabulary server 
program. By default, all words can be added as public, 
that is to be shared, but a user can choose to have add- 
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ed words held as private, if desired for any reason. 
[0040] A simple example has been charted in the Fig- 
ure. Client 14 has added a new word. The new word is 
transmitted over the network 20 from the add word pro- 
nunciation tool 34 to the word pronunciation server 12, $ 
where the new word is added to the universal vocabu- 
lary list 24, as indicated by dashed line arrow 1 . If client 
16 is dictating, or sometime thereafter begins dictating, 
the universal vocabulary server program 22 recalls the 
user profile for client 16 from the user profile table 26, 10 
as indicated by dashed line arrow 2. The profile reveals 
that client 16 belongs to the same workgroup as client 
1 4, represented by the use of the same profile A. If client 
1 6 is already dictating, the new word is delivered to client 
16 and added to the local cache 36. If client 16 begins is 
dictating, a personalized vocabulary list is delivered to 
client 16 and stored in the local cache 36. The new word 
added earlier by client 1 4 is included, because both user 
14 and user 16 use profile A. Both instances are indi- 
cated by dashed line arrow 3. 20 
[0041] In accordance with the inventive arrange- 
ments, a comprehensive database is dynamically and 
incrementally enhanced by users and automatically op- 
timized over time to increase overall system perform- 
ance and to balance computer system resources, such 25 
as network bandwidth and file and data storage, using 
caching techniques. 

[0042] In the present invention the central vocabulary 
list may comprise a plurality of data records for each 
word, said data records including: spelling; phonemes; 30 
usage domains; client of origin; and, workgroups requir- 
ing said word. 



Claims 35 

1 . A network spoken language vocabulary system for 
a speech application, characterised by a network 
server (12) and a plurality of network clients (14. 16, 
18) communicating with one another over a net- 
work: a central vocabulary list (24) in said server for 
recognizable words; and a speech recognition en- 
gine (32) and a local vocabulary list cache (36) in 
each of said clients, said cache having therein pre- 
viously recognized words communicated from said *s 
central vocabulary list in said server and new words 
recognized by said speech application; each of said 
new words being communicated to said server and 
added to said central vocabulary list and each of 
said new words added to said central vocabulary so 
list being communicated to at least one other of said 
clients for addition to said cache. 

2. A system of claim 1 , characterised in that said new 
words are automatically communicated back to said ss 
server. 

3. A network spoken language vocabulary system for 
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a speech application, characterised by a network 
server (12) having a permanent central vocabulary 
list (24) for recognizable words and a user profile 
data base (26): a plurality of network clients (14, 16. 
18), each having a speech recognition engine (32) 
and a local vocabulary list cache (36): and a net- 
work (20) over which said server and said clients 
can communicate: said cache (36) having stored 
therein a subset of previously recognized words 
communicated from said central vocabulary list (24) 
in said server (12) and new words as recognized by 
said speech application during said session; each 
of said new words being communicated to said 
server (12) and added to said central vocabulary list 
(24); and each of said new words added to said cen- 
tral vocabulary list (24) being communicated to oth- 
er ones of said clients in accordance with data in 
said user profile data base (26). 

4. A system of claim 1 or 3 : characterised in thai said 
new words are automatically communicated to and 
from said server. 

5. A system of claim 1 or 3, characterised by a user 
profile data base having data therein for each of 
said users indicative of a respective subset of said 
central vocabulary list for use in speech recognition. 

6. A system of claim 1 or 3 : characterised in that said 
user profile data base has data therein for each of 
said users indicative of membership in a work 
group 

7. A system of claim 6, characterised in that each new 
word communicated to said server from any user in 
one of said workgroups is automatically communi- 
cated back to all other users in said work group. 

8. A system of claim 1 or 3 : characterised in that said 
user profile data base has data therein for each of 
said users indicative of membership in a work 
group : each user in said work group receiving the 
same subset of said central vocabulary list for use 
in speech recognition. 

9. A system of claim 1 or 3, characterised in that each 
of said clients further comprises an add word pro- 
nunciation tool for identifying said new words to be 
communicated to said server, and for adding said 
new words to said cache. 

10. The system of claim 9, characterised in that said 
add word pronunciation tool adds said new words 
as recognized by said speech recognition engine in 
said client and as communicated from said server. 
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(54) Extending the vocabulary of a client-server speech recognition system 



(57) A network spoken language vocabulary system 
for a speech application comprises: a network server 
(12) and a plurality of network clients (14, 16, 18) com- 
municating with one another over a network; a central 
vocabulary list (24) in the server for recognizable words; 
and a speech recognition engine (32) and a local vocab- 
ulary list cache (36) in each of the clients. The cache 
(36) has therein previously recognized words commu- 



nicated from the central vocabulary list (24) in the server 
and new words recognized by the speech application. 
Each of the new words is communicated to the server 
and added to the central vocabulary list and each of the 
new words added to the central vocabulary list is com- 
municated to at least one other of the clients for addition 
to the cache. The new words can be automatically com- 
municated to and from the server. 
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